Despite numerous advances in tools to assist the creation of cinematography, it remains a time-consuming and largely manual endeavour requiring knowledge of established filmmaking conventions as well as creativity in knowing how to communicate a visual message using a camera. A cinematographer must consider the appropriate positioning, aiming, lens choice, movement of the camera and edits to visually communicate a story entailing compliance with conventions in screen composition, viewpoint selection and continuity rules. Such conventions guide the way successive viewpoints need to be arranged in time to effectively present a sequence of images.
These same issues must be addressed when producing cinematography in computer-generated virtual worlds by positioning and aiming a mathematical model of a virtual camera. Attempts to simplify the production of cinematography, especially for novices, include embedding composition guides into physical hand-held digital cameras, automated virtual camera planners, and physical devices equipped with motion sensors. However, each of these existing methods leaves much room for improvement.
Digital camera composition guides: Some consumer-level hand-held digital cameras include limited facilities to assist a user in composing a photograph as he or she peers through the viewfinder. Features include detecting faces and automatically adjusting focal depth so that the faces appear in sharp focus, pre-defined camera settings optimized for various common shooting scenarios such as night time, and grid lines to assist in aligning subjects within the frame. These solutions only assist in composing one single image as currently seen through the viewfinder and deal with only the most rudimentary composition guidelines.
Automated virtual camera planners: The research field of automated virtual camera planning, which combines expertise in Computer Graphics and Artificial Intelligence, has resulted in increasingly sophisticated algorithms to emulate the composition and editing decisions found in traditional films. Early efforts in automated camera control employed hand-coded camera locations relative to the subject(s) being viewed while attempts to model the editing process have relied on idiom-based approaches using hierarchical state machines to model and transition between commonly used shot types [Li-Wei He et al. 1996]. Existing implementations of the idiom-based approach require extensive human effort to design finite state machines to encode the shot selection and transition logic needed to film a given anticipated type of scenario such as groups of two or three characters conversing. Heuristic search-based or optimization techniques repeatedly generate and evaluate the quality of candidate viewpoints until a satisfactory one is found [for example Drucker and Zeltzer 1995—“CamDroid: A System for Implementing Intelligent Camera Control,” in Proc of the 1995 Symp. on Interactive 3D Graphics, pp. 139-144, 1995]. This approach is generally effective in producing acceptable cinematography when characters vary somewhat in their position, orientation, or size. However, if a filmmaker wishes for the system to produce a specific viewpoint in a given situation, then he or she must adjust the parameters that govern the optimization such as changing evaluation scoring weights, the constraints, and/or initial values of the search. Once the optimization search begins, it is largely out of the user's control, until the solver reports its solution. Consequently, a user who is not intimately familiar with the inner workings of the optimizer could become frustrated trying to coax the optimizer to output the desired virtual camera composition. Even worse, these techniques have found little to no use in producing real-world cinematography primarily because they require specialized expertise to implement and operate, and their results, which are often excellent in mimicking anticipated sequences of textbook cinematography, fail to produce the type of creative results demanded by expert human filmmakers. In attempting to minimize the burden of human input to manually control a virtual camera, these methods have consequently neglected to consider novel ways of incorporating creative human input into the automated camera planning process.
Motion-sensing virtual cameras: Some movie production and virtual reality companies have developed motion-tracked virtual camera devices (FIG. 3) that couple camera-like bodies or camera mounts, preview monitors, and usually buttons and joysticks with 3DOF (orientation) or 6DOF (position and orientation) motion sensors and software plug-ins that map device inputs to virtual camera parameters in high-end animation packages (e.g. Autodesk's MotionBuilder, 3DS Max, or Maya). With a 6DOF (position and orientation) sensing system, a camera operator can simply walk about and point the virtual camera device in the most intuitive and familiar way possible, which in turn produces correspondingly similar moves of the virtual camera. These solutions have in effect replaced the mouse and keyboard with a more intuitive input device, but they have not considered how the user interface and workflow of the accompanying computer-generated (CG) animation software may be enhanced to more fully take advantage of these motion-sensing inputs. These solutions do not allow a user to easily create, organize, and consider side-by-side alternative ways to film a given sequence of events. Their users must still expend considerable time and energy moving and turning within a volume (sometimes up to 40×40 feet or larger) of real-world space to explore alternate viewpoints.
U.S. Pat. No. 6,040,841 discloses a virtual cinematographer.
Thus there is a need to provide for an easier and richer method for creating and editing camera views and movements which dynamically integrates creative human input into automated virtual cinematography tools.